Automated cytological specimen classification system and method

ABSTRACT

An automated screening system and method for cytological specimen classification in which a neural network is utilized in performance of the classification function. Also included is an automated microscope and associated image processing circuitry.

This application is a continuation of application Ser. No. 08/486,961filed Jun. 7, 1995 now U.S. Pat. No. 5,740,270 which is a continuationof application Ser. No. 07/502,611 filed Mar. 30, 1990 (abandoned) whichis a continuation-in-part of U.S. patent application Ser. Nos.07/179,060 and 07/420,105 (now U.S. Pat. Nos. 4,965,725 and 5,287,272 )and U.S. patent application Ser. No. 07/425,665 (abandoned) filed Apr.8, 1988, Oct. 11, 1989 and Oct. 23, 1989, respectively, entitled“Automated Cytological Specimen Classification System and Method”, theentire disclosures of which are hereby incorporated by reference.

TECHNICAL FIELD

This invention relates generally, as indicated, to cell classificationand, more particularly, to a system for increasing the speed andaccuracy of cervical smear analysis.

BACKGROUND OF THE INVENTION

The examination of a cervical smear by what often is referred to as aPap test is a mass screening cytological examination which currentlyrequires individual visual inspection by a person of virtually all ofthe approximately 100,000 cells on a typical slide. The test, therefore,suffers from a high false negative rate due to the tedium and fatigueassociated with this requirement for exhaustive search.

Prompted by the clear commercial potential for automated cervical smearanalysis, several attempts to this end have been made heretofore. Theseattempts have proven to be unsuccessful at least partly because theycould not accommodate overlapping cells as are typically found in thePap smear. To circumvent the classification problems created byoverlapping cells, specialized “monolayer preparations” have beenprepared. A monolayer preparation is a specially prepared smear inwhich, the cervical cells are centrifuged and filtered so that only asingle layer of cells results. Besides serious cell preservation andcell transportation problems, the expense and time involved in themonolayer preparation precludes its use as a population screeningsubstitute for the Pap smear.

Even when limited to the non-overlapping cell images provided by themonolayer preparation, prior art attempts at automated cytologicalclassification have not been able to process cervical smear images atanything close to manual processing time. Many of these attempts atautomated cytological classification have relied on feature extractionalgorithms which attempt to select and to measure some feature withinthe image, e.g., the shape of the cell nucleus. Feature extractionalgorithms have failed due to the inability to segment the image intothe components which require measurement. One cannot measure nuclearsize, for example, unless the image is segmented so that the cellularnuclei are identified. Template matching, in which an actual image (nota mathematical quantity) is compared with stored exemplar images alsohas not been successful since it is computationally intensive and theinfinite variety of possible Pap smear images or scenes would require anexcessive number of exemplar image comparisons. The distinction betweenfeature extraction and template matching is outlined in the Collingsreference on pages 1 through 5 while image segmentation techniques arediscussed in Chapter 7 of the Gonzalez reference.

An example of the limitations of the prior art can be found in the 1987reference entitled “Automated Cervical Screen Classification” by Tien etal, identified further below.

Background references of interest are, as follows:

Rumelhart, David E. and McClelland, James L., “Parallel DistributedProcessing,” MIT Press, 1986, Volume 1;

Tien, D. et al, “Automated Cervical Smear Classification,” Proceedingsof the IEEE/Ninth Annual Conference of the Engineering in Medicine andBiology Society, 1987, p. 1457-1458;

Hecht-Nielsen, Robert, “Neurocomputing: Picking the Human Brain,” IEEESpectrum, March, 1988, p. 36-41; and

Lippmann, Richard P., “An Introduction to Computing with Neural Nets,”IEEE ASSP Magazine, April, 1987, p. 4-22.

BRIEF SUMMARY OF THE INVENTION

An object of the invention is to automate at least part of theclassification procedure for cytological specimens.

Another consistent objective is to provide semi-automation in acytological specimen classification apparatus and method, whereby atleast part of the cell classification procedure may be carried out by ahuman being.

Consistent with the foregoing, an object of the present invention is toclassify cytological specimens into categories, for example, categoriesof diagnostic significance, and, more particularly, to automate at leasta part of such classification procedure.

As used herein the term “automated” means that at least part of theapparatus is automated; in the preferred embodiment a portion of themethod is carried out by a person.

Briefly, according to one embodiment, the invention includes an initialclassifier (sometimes referred to as a primary classifier) preliminarilyto classify a cytological specimen and a subsequent classifier(sometimes referred to as a secondary classifier) to classify thoseportions of the cytological specimen selected by the initial classifierfor subsequent classification.

According to one embodiment, the invention includes an initialclassifier (sometimes referred to as a primary classifier) preliminarilyto classify a cytological specimen, a subsequent classifier (sometimesreferred to as a secondary classifier) to classify those portions of thecytological specimen selected by the initial classifier for subsequentclassification, and a tertiary classification to determinecharacteristics of or to classify those portions of the cytologicalspecimen that are selected by the subsequent classifier for furtherclassification.

In one embodiment the primary classifier performs a low levelmorphological feature screening function on the entire image while thesecondary classifier performs a high level pattern matchingidentification on those images not screened out by the primaryclassifier.

In an embodiment of the invention the primary classifier classifiesspecimens according to size criteria and integrated optical density.

In an embodiment the secondary classifier is a neural net.

In an embodiment the tertiary classifier may be a person.

In an embodiment the present invention performs its classification of agroup of specimens within the period of time typically consumed for thistask by careful manual screening (i.e., approximately sixminutes/specimen) or faster.

In an embodiment the present invention performs its classification oncytological specimens which contain the numbers and types of objectsother than single layers of cells of interest that are typically foundin cervical Pap smears (e.g., clumps of cells, overlapping cells,debris, clumps of leucocytes, bacteria, mucus) .

In an embodiment the present invention performs the above-describedclassification on cervical smears for the detection of pre-malignant andmalignant cells.

In an embodiment the present invention displays, e.g., on a monitor orother display medium, cells adjacent or near one or more exemplary cellshaving features distinctive of a certain cell classification, such aslarge dark nuclei for malignant or pre-malignant cells, to facilitate,by comparison, cell screening by a person.

In an embodiment the present invention performs its classification withsmaller false negative error rates than those typically found inconventional manual cervical smear screening.

In an embodiment of the present invention classification of cytologicalspecimens into medically significant diagnostic categories will be morereliable, i.e., will have lower false negative error rates, than presentmethods.

In an embodiment the cytological classification system of the presentinvention does not require a modification in the procedure by whichcellular specimens are obtained from the patient, i.e., standard Papsmears are used for its input.

In an embodiment the cytological classification system of the presentinvention will permit reliable classification within processing timeconstraints that permit economically viable operation.

In an embodiment of the invention classification of a cytologicalspecimen is made by a person, and subsequent automated (orsemi-automated) classification of selected specimens, such as thoseprimarily noted as negative by such person, or, if desired, of allspecimens, then may be carried out.

In an embodiment an automated specimen transfer mechanism is provided totransport cytological specimens between a storage location and anexamination location.

In an embodiment of the invention a marking system marks selected areasof a cytological specimen at which prescribed characteristics appear,such marking being either automatically, semi-automatically or manuallyinitiated.

In an embodiment of the invention classification of a cytologicalspecimen is authorized if an authorized identification is associatedwith the specimen, and such classification may be prevented if suchauthorized identification is not found.

In an embodiment of the invention improvements are provided to anautomated microscope, including, for example, one or more of focuscontrol, light intensity control, positioning control, and lens orobjective magnification changing.

In an embodiment of the invention the reliability of a cytologicalscreening test is determined by ranking cells found in the specimenbased on the degree to which they exhibit features typical ofpremalignant or malignant cells, and then examining at least some ofthose cells for the presence of a cell type which indicates that thespecimen was obtained properly.

In an embodiment of the invention the determination of the presence ofan infection in a cytological specimen includes ranking the cells foundin the specimen based on the degree to which they exhibit featurestypical of a premalignant or malignant cell, and examining at least someof the ranked cells for cells having indications of infection.

These and other objects, advantages and features of the presentinvention will become evident to those of ordinary skill in the artafter having read the following detailed description of the preferredembodiment.

It is noted here that the published articles cited herein arespecifically incorporated by reference.

Moreover, it is noted here that the invention is described herein mainlywith respect to classification of cytological specimens in the form of acervical smear, as typically is done in connection with a Pap test.However, it will be appreciated that this is but one example of theapplication of the principles of the invention which may be used toclassify other cytological specimens.

BRIEF DESCRIPTION OF THE DRAWINGS

In the annexed drawings:

FIG. 1A is a schematic illustration of an automated cytological specimenclassification system according to the invention;

FIG. 1B is a schematic illustration of the automatic fine focusadjustment of the system of FIG. 1A;

FIG. 2 is a block diagram of an automated cytological specimen screeningdevice in accordance with the present invention with particular emphasison classification components;

FIGS. 3A, 3B and 3C present a schematic flow chart short-handrepresentation of the method of classifying objects in an exemplaryoperation of the invention;

FIG. 4 is a block diagram representation of the method of cellclassification used by the device of FIG. 1;

FIG. 5 is a block diagram of a classification of a slide having nopathological cells; and

FIG. 6 is a block diagram of a classification of a slide having 50pathological cells.

DESCRIPTION OF THE PREFERRED AND ALTERNATE EMBODIMENTS

With reference to the drawings in which like reference numerals depictlike items, and specifically to FIG. 1A, there is illustrated a neuralnetwork based automated cytological specimen classification (sometimesreferred to as screening) screening device or system 10 in accordancewith the present invention. The screening device 10 includes anautomated microscope system 12, a camera 14, a barcode reader 16, aslide marker 18, and a computer processing system 20.

Briefly, the screening device 10 is used to classify cytologicalspecimens, in the preferred embodiment to determine and/or to help todetermine whether a cytological specimen contained on a slide S (or onor in some other support, container, etc.) includes characteristics orfeatures of interest. Exemplary characteristics are those which are hadby malignant or pre-malignant cells in what is commonly known as a Papsmear. In an exemplary embodiment described in detail below, theautomated microscope system 12 makes a low resolution examination of thespecimen during which some working parameters, such as the location ofthe specimen on the slide, focus, and/or illumination level (for optimumviewing in the subsequent high resolution examination), are determined.Authorization to conduct the examination, e.g., using the barcode reader14 to sense whether the barcode on a slide is proper, also may bedetermined prior to or during the low resolution examination.Thereafter, with reliance on such working parameters and authorization,a high resolution examination of the specimen is made by the microscopesystem 12. Based on information obtained during such high resolutionexamination, primary, secondary and tertiary classification proceduresare carried out to determine, for example, whether malignant orpre-malignant cells are contained in the specimen. Preferably theprimary and secondary techniques are automated and the tertiaryclassification is carried out manually, i.e., by a person. However,consistent with the invention other types of primary, secondary and/ortertiary image processing techniques may be employed.

Operation of the system 10 is generally under the control of thecomputer system 20. Accordingly, such computer system includes a generalpurpose computer 20 a, such as an AT type microcomputer, and an imageprocessor 20 b, such as one sold under the U.S. registered trademarkPIPE by ASPEX INCORPORATED, and a neurocomputer 82, such as that soldunder the U.S. registered trademark ANZA PLUS bh HNC, Incorporated.

The automated microscope system 12 includes a number of elementsdesigned to facilitate the quick and easy handling of specimen slides.One such element is a robotic slide handler 22 which, upon appropriatecommands from the computer system 20, moves the specimen slides from aholder 23, called a cassette, to a notorized movable stage 24 fortransport into and within the optical path of the microscope for cellclassification, and then back to the cassette after classification.Mounted to (preferably bolted to) the motorized stage 24 is a slidesupport bridge (sometimes referred to as a tooling plate or toolingfixture) 25 upon which the slide is held for movement through themicroscope system 12. The bridge 25 has one or more passages 26(preferably four) in the area below where the slide is positioned andopening to face the slide to allow the creation of a vacuum under theslide to hold it firmly down upon and in place on the bridge. Thetooling fixture bridge 25 also includes a relief area or slot 27 toallow the robotic slide handler 22 to grip the slide during placement onand removal from the tooling fixture bridge 25. The slot 27 alsoprovides space for illuminating light from a light source 28 to travelalong an optical path designated 29 to the microscope objectives 30, 31.

The motorized stage 24 is mounted upon cross roller bearings 32 andpowered by two stepping motors 33, 34 with associated drivers byCompumotor together with a Parker Compumotor PC23 interface controller35 to provide precise movement of the slide relative to the optical path29 and viewing area of the microscope. The bearings 32 are mounted tothe microscope base or frame 40, which in turn is attached byconventional springs and vibration damping shock absorbers 41 to a heavy(e.g., 500 pounds) granite base 42. Support of bearings 32 also may beby springs and/or shock absorbers 43.

The stage motors 33, 34 are controlled by the motor controller 44 (thementioned PC23 interface) which converts commands from the computerprocessing system 20 into appropriate signals to cause the motors 33, 34to move the stage to the prescribed locations. The precise location ofthe stage 24, and thus the slide, is detected by the position encoder44. Such position encoder 44 may be a conventional device that producespulses representative of movement or location of the stage. Thosepulses, as is conventional, may be decoded by the computer system 20 toidentify the current location of the stage 24, e.g., relative to a homeor reference location or relative to some other location. An exemplaryposition encoder is sold by Heidenhain.

The automated microscope 12 also includes features to provide fast,precise imaging of each area and/or of selected areas of a slidepositioned on the bridge 25, as is described further below. The opticalsystem 45 of the automated microscope 12 includes an objective carriage45, and an autofocusing mechanism 46. The light source 28 includes alamp 47 of constant color temperature, light intensity control filters50, an automated iris or diaphragm 51, and associated reflectors,prisms, lenses, light conductors, etc., schematically represented at 52,to send light along the optical path 29 to illuminate the specimen frombelow. (If desired, illumination of the specimen can be from above.)

The objective carriage 45 moves the appropriate magnification objective30 and 31 (say, respectively, of 5× and 20× magnification), or lenssystem, into place in the optical path 29 to provide for low or highresolution viewing of the slide, respectively, as desired during theparticular phase of operation. A motor 53 which is controlled via aconnection 54 by the computer system 20 selectively moves the carriage45 and respective objectives 30, 31 into the optical path 29. Limitswitches 55, 56 sense maximum travel of the carriage 45 and cooperatewith the computer system 20 in standard fashion to prevent over-travelof the carriage. A conventional adjustment 57 is schematically shown forparcentering, i.e., centering of the respective objectives 30, 31 inoptical path 29 regardless of whether the carriage 45 has moved toposition one or the other objectives in the optical path.

The autoiris 51, such as an adjustable aperture or diaphram, is forcontrolling the intensity of light transmitted to the slide and in theoptical path 29 of the microscope 12 to the objectives 30, 31. Theautoiris automatically adjusts the transmitted light intensity accordingto which objective 30, 31 the carriage 45 has positioned in the opticalpath 29. A motor 60 controlled by the computer system 20 adjusts theiris 51 to respective relatively more open and relatively more closedconditions at the same time that motor 53 moves the carriage 45 to placerespective low and high resolution objectives in the optical path 29.

The filters 50 may be counter rotating variable neutral density linearpolarizer filters 50 a, 50 b positioned in the optical path 29 betweenthe light source 41 and the iris 51 to provide further control of thelight intensity transmitted into the optical path without affecting thecolor temperature of the light. A motor 61 rotates the filters 50 a, 50b under control of computer system 20, which may automatically call foran increase or a decrease in slide illumination intensity, e.g., asrequested by a user, as the light source ages and/or is changed, etc. Byrotating the polarizers 50 a, 50 b, the extent that they cross or areparallel and, thus, the amount of transmission therethrough can becontrolled.

The automated microscope includes a coarse focus adjustment mechanismand a fine focus adjustment mechanism 69 (FIG. 1B), both being ofconventional design, and, therefore, neither of which is shown indetail. The coarse focus adjustment may be operated by a micrometer typecontrol that can be operated, e.g., turned manually. The fine focusadjustment also can be operated manually if desired. However, the finefocus adjustment preferably is operated automatically by the autofocus46 in the manner descried below. (Other types of focus adjustments alsomay be employed.)

As is seen in FIG. 1B, the autofocus 46 includes a light source 70, abi-cell 71 (a device with two photocells or other photosensitive devices72, 73 in a housing 74 with a mask formed by a pair of openings 75, 76for guiding light to the respective photocells,) a differentialamplifier 77 with an offset connection to the computer 20, apiezoelectric device 78, and a mechanical coupling 79 to the fine focus69. Light from the source 70 is reflected off the top surface of thecover slip S′ on slide S. The photosensors 72, 73 produce electricaloutputs representing the position of the slide relative to the lightsource and the bi-cell 71. Differential amplifier 77 determines thedifference between the photosensor outputs (it being desirable that suchdifference be minimized). An offset voltage can be provided via thecomputer and a digital to analog converter (not shown) to compensate forthe thickness of the cover slip S′ so that the actual point of focus isat the surface of the slide S or a desired depth into the specimen onthe slide S. The output from the differential amplifier 77 may befurther amplified by an amplifier 77 a, and the output from suchamplifier is used to provide a voltage input to the piezoelectric device78. The piezoelectric device then mechanically operates the fine focus69 control of the microscope 12 via the mechanical connection 79.

Another focus control for the microscope 12 also may be provided. Suchfocus control may rely on image processing, as is described furtherbelow, to determine the degree of focus that an object is seen by thecamera 14 in particular. Such image processing focus control can be usedboth to make a focus map of the slide S and also to control the finefocus adjusting mechanism 69 and/or the coarse focus adjusting mechanismto bring the image into focus for the camera.

The image processing focusing described may be carried out during highand low resolution viewing. During low resolution viewing focalinformation is correlated with position coordinates from the positionencoder 44 to compose a focal map, and the resulting focal map is storedin computer memory. During high resolution viewing, the computer system20 provides the focal information from the stored focal map inaccordance with the location of the viewing field as determined by theposition encoder 44 to adjust focus by mechanically controlling thecoarse and/or fine focus adjustment mechanisms of the microscope 12.This allows for fast focusing during the high resolution scan.

Alternatively, focusing can be carried out during other portions of thelow and/or high resolution examinations of the slide. As an example, thefocusing function can be carried out prior to the initial low resolutionexamination of the slide and/or prior to the first high resolutionexamination of the slide. In the high resolution examination, as isdescribed further below, several specific areas or “tiles” of the slideare examined, and focusing function can be carried out before each tileis examined or after the initial focusing such focusing can be carriedout each time a predetermined number, e.g., five, of tiles has beenexamined.

The barcode reader 16 may be a conventional device, such as an integralbarcode reader system, which is sold by Symbol Technologies, Inc ofBohemia, N.Y. under the trademark LaserScan 6X20. Such barcode reader 16is positioned to view a selected area of a slide once it has beentransported to the stage 24 by the robotic slide handler 22. In thepreferred embodiment each slide presented to the system 10 forclassification must contain a barcode. The barcode contains relevantinformation necessary in coordinating the classification results to theslide, e.g., the doctor providing the slide and the patient from whichthe specimen on the slide was obtained. The barcode reader 16 reads theencoded information from the slide and provides that information to thecomputer system 20 via connection 80 for storage and future correlationwith test (i.e., classification) results. In the event that a slide ispresented to the screening system 10 without a barcode, or with animproper or unreadable barcode, the slide will be rejected and returnedwithout classification to the cassette 23 by the robotic slide handler22. In other embodiments the barcode and barcode reader could bereplaced by a system performing similar functions, such as a set ofcharacters and an optical character reader.

When a physician collects a cytological specimen from a patient, thephysician may securely affix a barcode label (e.g., an adhesive label)to the slide on which the specimen is placed and may securely affix acorresponding label and/or barcode to the patient's chart. The chart maybe retained by the physician. When the system 10 examines a slide,preferably it also prints a report of the results of such examination.The barcode information read by the system 10 preferably is printed onsuch report. Then the physician can compare the printed barcodeinformation with the corresponding barcode or like information on thepatient's chart to confirm accurate matching of the report to thepatient. The physician also can supply the laboratory, which is usingthe system 10 to classify the cytological specimen, written information,such as the patient's name and the physician's name, that can becorrelated with the barcode and printed automatically on the reportconcerning the classification specimen.

The camera 14 is positioned in the microscope's optical path 29 tocapture a focused, magnified electronic image of the area of the slidebeing viewed. The camera 14 feeds the electronic image to the computersystem 20 via a connection 81 for classification of the cells appearingin the imaged area. The camera may be a conventional three chip chargecoupled RGB camera, such as one made by Sony, or other camera able toprovide suitable information of the specimen, i.e., an image, to thecomputer. Such image is preferably represented by electrical signalswhich can be processed by the image processor 20 b of the computersystem 20.

In the computer system 20 a number of image processing and evaluationfunctions are performed. These include determining where on a slide isthere specimen material actually located, whether there is adequatespecimen material to perform a meaningful classification, and theprimary classification which does a low level filtration or screening,e.g., based on morphology that can be evaluated algorithmically. Theneurocomputor portion 82 of the computer system 20 provides thesecondary classification, doing a higher level filtration or screeningbased upon training of the neurocomputer, e.g., as is disclosed in theparent patent application mentioned above and the references mentionedabove as well as according to the description presented herein.Electronic image representations of cells which are classified by theprimary and secondary classifiers as being suspect are stored in thecomputer memory, in disk memory, or in some other mass storage devicefor further (tertiary) classification by a person trained to detect thetruly abnormal cells. The locations of such suspect cells on the slidealso are stored in the computer. Subsequently, when the tertiaryclassifier (technician) can review the images of such suspect cells,they can be inspected or examined by viewing such stored images on avideo monitor; and the technician can make a final determination as towhether each of such suspect cells is truly abnormal, e.g., malignant orpre-malignant or otherwise of interest.

When the tertiary classifier finds such an abnormal cell, she or he mayuse a mouse or some other device coupled to the computer 20 a to pointto such cell. Such cell then is identified or is tagged (e.g., insoftware or an electronic data representation of such cell) forconvenient recall and redisplay for viewing and examination by asupervisor, pathologist, etc. An image of such cell may be printed ontopaper. Also, the physical location of such cell on the slide can bemarked, e.g., by placing an ink dot proximate such cell on the slide oron the cover slip thereof.

More particularly, after the tertiary classifier has identified thetruly abnormal cells, the computer system 20 commands the motor control35 and stage motors 33, 34 to position the respective areas of the slidehaving an abnormal cell under the slide marker 18. (If the slide hadbeen returned to the cassette 23, the robot handler 22 first replacesthe slide back on the bridge in the same position as it had been beforewhen inspected by the camera and so on.) The slide marker 18 then iscontrolled by the computer 20 to make a small dot, approximately 0.25 mmin diameter, on the slide in the (several) area(s) of the slide wheresuch abnormal cell(s) is (are) located by operation of a solenoid (notshown) to dab ink via an arm 18 a onto the slide. Such marking issimilar to the manner in which silicon wafers are marked for specifiedpurposes in the field of such wafer manufacturing and inspection. Thecomputer system 20 will then command the robotic slide handler 22 toreturn the classified slide to the cassette 23. Another slide then canbe marked as described until all slides in the cassette have beenscreened. The slide marker may be one of the type sold by Xandex.

Motion of the stage 24 to bring portions of a slide S that is held onthe bridge 25 is under control of the computer 20. The computer sendscontrol signals to the interface control circuit 35, which in turn sendsappropriate signals to driver circuits associated with the steppermotors 33, 34 to move the stage in the X and Y directions indicated inFIG. 1. Limit switches 90, 91, 92, 93 detect maximum travel of the stageand are coupled to the computer 20 to limit travel beyond maximumlimits, as is conventional. Feedback to indicate the actual location ofthe stage or the relative location compared to a reference location isobtained by the position encoder 44, as is conventional.

The actual travel path of the stage to move the slide to desiredlocations relative to the optical path 29 and/or to move the slide to aposition convenient for pick up and placement by the robot handler 22can be programmed into the computer using conventional techniques. Themoving of the stage 24, bridge 25 and slide S relative to the opticalpath 29 to place specific cells into the optical path for viewing duringrespective low and high resolution examinations by the respectiveobjectives 30, 31 and when specific cells are desired to be reexaminedalso can be controlled automatically as a function of locationinformation that can be stored in the computer and/or mass storagememory. Further, the locations to move cells for automatic dotting bythe marker 18 and arm 18 a also can be controlled automatically based onstored location information that is delivered to the computer 20.Appropriate offsets may be added to location information, e.g., to movea cell desired for marking relative to the marker arm 18 a taking intoconsideration the location of the marker arm 18 a, the location of theoptical path, and the desire to place the mark adjacent the cell (andnot directly on the cell so as to obscure the cell in case it is to beviewed subsequently).

Considering, now, the robotic slide handler 22, such device is similarto a silicon wafer robotic handler used in the semiconductor industryfor automatically handling, moving, etc., silicon wafers on whichintegrated circuits or the like are formed. Accordingly, such handler 22includes an arm 100, which is mounted relative to the microscope frame40 for movement to move slides from the cassette 23 to the stage 25 andvice versa. Motive means 101 for mounting and moving such arm areprovided in a support housing 102, Appropriate joints, etc., may beprovided in the arm 100, depending on the degrees of freedom of movementdesired. At the end of the arm 100 is a support foot 103 on which aslide actually is carried.

The foot 103 includes a top surface 104 that can fit beneath a slide S,which is shown in the cassette. An opening 105 in the surface 104 of thefoot and a passage to such opening provides a source of vacuum to holdthe slide S to the foot. The vacuum opening is coupled via a passage 106and valve 107 to a vacuum source 110. Such vacuum source 110 also iscoupled via vacuum line 111 and valve 112 to the vacuum lines andopenings 26 in the stage 25.

The cassette 23 is mounted on an elevator 113 and preferably is heldthereon by a vacuum drawn through openings 114, vacuum line 115 andvalve 116. The valve 116 preferably is manually controlled (energized ordeenergized) by a manually operated electrical switch 117. Closure ofsuch switch 117 energizes the valve 116 to provide a vacuum that holdsthe cassette onto the elevator 113; opening of the switch deenergizesthe valve to release the cassette from the elevator enabling easyremoval therefrom.

The elevator 113 includes a lift mechanism 120 and a support platform121. An electronic control circuit 122, which can be controlled by thecomputer 20, provides information to the control circuit 122 to identifyfrom where to obtain the next slide (e.g., from the cassette and wherefrom the cassette or from the stage 25) and to where to move the slide.The control circuit 122 may be of the conventional type used in robotsystems that handle silicon wafers, as was mentioned earlier. Thus, forexample, upon sending the appropriate signal to the control circuit 122,the computer can initiate an operational cycle of the handler 22, say toremove a slide from the cassette 23 and to place the slide on the stage25, and vice versa.

Coordinated operation of the arm 100, foot 103, elevator 113, stage 24,and control circuit 122 is controlled by the computer 20. As oneexample, to pick up a slide S from one particular location of many slidestorage locations in the cassette 23, the elevator 113 lifts thecassette 23 slightly to provide clearance for the foot 103 beneath theparticular slide S. The motive mechanism 101 moves the robot arm 100 andfoot 103 to place the foot beneath the slide. The valve 107 then isenergized by the control circuit to supply vacuum to the opening 105.Then the elevator 113 lowers the cassette 23 to lower the slide S ontothe top surface 104 of the foot 103. The slide then is held onto suchsurface 104 by the vacuum and by gravity.

The computer by now has caused the stage 24 to move the bridge 25 toslide loading position. The arm 100 then swings to move the slide S intoalignment above the openings 26 at an appropriate location on the bridge25. The bridge 25 and/or the foot 103 are moved vertically relative toeach other so that the slide is placed onto the top surface of thebridge 25. The foot 103 fits in a recess 27 in the bridge. The valve 112is energized to supply vacuum to the openings 26 to hold the slide onthe stage, and the valve 107 is deenergized to release the vacuumholding the slide to the foot.

Similar operation can be used to move the slide from the bridge 25 backto the cassette, and so on.

Referring to FIG. 2, the screening device 10 is shown with particularemphasis on the classification elements embodied in the computer system20. The computer system 20 includes an image processor and digitizer 20b, a neurocomputer 82, an output monitor 154, and a general processor 20a with peripherals for printing, storage, etc.

The general processor 20 a is preferably an IBM PC/AT or compatiblealthough it may be another computer-type device suitable for efficientexecution of the functions described herein. The processor 20 a controlsthe functioning and the flow of data between components of the device10, causes execution of additional primary feature extraction algorithmssuch as an integrated optical density function and handles the storageof image and classification information. The processor 20 a additionallycontrols peripheral devices such as a printer 158, floppy and hard diskstorage devices 160, 162, and the barcode reader 16, slide marker 18,autofocus 46, robotic slide handler 22, stage motor controller 35, andobjective carriage 45 components described more fully above.

The image processor and digitizer 20 b also performs primary cellclassification functions such as thresholding and erosion and dilation.In the preferred embodiment, the image processor and digitizer 20 b is acommercially available low level morphological feature extraction imageclassifier such as the ASPEX Incorporated PIPE (TM) image processorwhich includes an image digitization function. The PIPE (TM) imageprocessor is described more fully in U.S. Pat. No. 4,601,055, the entiredisclosure of which hereby is incorporated by reference. Alternatively,the image processing and digitization function could be separated intotwo or more components.

Secondary group processing cell classification is performed by theneurocomputer 82. The neurocomputer 82 is a computer embodiment of aneural network trained to identify suspect cells. In this embodiment theparallel structure of a three-layer backpropagation neural network isemulated with pipelined serial processing techniques executed on one ofa host of commercially available neurocomputer accelerator boards. Theoperation of these neurocomputers is discussed in the Spectrum referencecited. The neural network is preferably implemented on an Anza Plusprocessor, which is a commercially available neurocomputer of ScienceHecht-Nielsen Neurocomputers (HNC) (see the Hecht-Nielsen referenceabove). Alternatively, secondary cell classification functions could beperformed using a template matching algorithm designed to identifyshapes known to be typical of a pathological cell. A template matchingor other group processing algorithm could be efficiently implemented ina parallel distributed processing network, for example. Anotheralternative secondary classification embodiment is a holographic imageprocessor designed to perform group based classification.

Referring to FIGS. 3A and 3B a flow diagram of the operation of the cellclassification method of the present invention is outlined.Initialization of the system 10 is carried out at block 199. Duringinitialization the image processor 20 b is initialized so as to be readyto receive the first electronic information representing the imagereceived by the camera 14 and to process the image information. Also thestage 24 is initialized to place it in a reference location, sometimesreferred to as a home location so that the computer 20 can expect toknow that the stage is at such location and can determine futurelocations based on the home location. The robot handler 22 isinitialized, too, to place the various portions thereof in a homeposition to so that future locations and movements thereof can bedetermined from the initial home position. The neural net also isinitialized in conventional fashion.

To begin operation, the robotic slide handler 22 grabs the first slidefrom the cassette 23 and transports it to the motorized stage 24 (block200). If the robotic slide handler was unable to find the next slide,such as when all slides in the cassette 23 have been classified, amessage is conveyed to the operator (205). Assuming a slide is present,the barcode reader 16 reads the barcode information from the slide (210)and passes the information to general processor 20 a (215) forcorrelation with future classification data. Cells not having barcodesor barcodes that are not readable are rejected and the next slide isprocessed (216). Next, the general processor 20a commands the stagemotor controller 35 and motors 33, 34 to move the stage 24 and slideinto the optical path 29 of the microscope system 12 for the lowresolution pass (220).

In the low resolution pass, the carriage 45 will move the low resolutionobjective 30 into the optical path 29, and the autoiris 51 willautomatically adjust the lighting for the low resolution objective(225). A relatively quick scan of the slide is then made to find theareas of the slide having cellular matter (230). If the no areas havingcellular matter or an adequate amount of matter for valid classificationis found on the slide, then the slide is identified as containinginsufficient cellular matter to perform a meaningful test (235); theslide then may be rejected (240) and screening of the next slide maybegin (200).

To determine whether there is adequate cellular material on the slide orat various locations on the slide and also to determine the focus mapfor the slide, the following may be carried out by the computer system20. First, a scan route is determined so that a plurality of areas onthe slide can be viewed sequentially. Such areas may be located in astraight line along the length of the slide S or in some otherarrangement on the slide. As an example, a plurality of areassequentially located along a serpentine path along the slide are viewed.Each such area will be designated hereinafter a “macro-tile”.

When a particular macro-tile is located in the optical path 29, thecamera 14 takes a picture thereof. The macro-tile may be, for example, 2mm by 2 mm in size. In the image processor 20 b, using an ISMAP programor algorithm available from ASPEX Incorporated, New York, N.Y., (iconicto symbollic mapper) the macro-tile is subdivided into sixteen areasreferred to below as “tiles” and a sharpness image and a gray scaleimage are made. These images are used to determine whether there isand/or how much there is of cellular material in the macro-tile. Ahistogram of the absolute value of the difference between such imagesmay be used in the focus function of the computer 20 to determine afocus map for the slide.

More particularly, the value of the optical transmission characteristicsof the macro-tile is made; and simultaneously two successive 3 by 3Gausian filtrations are made to provide a 5 by 5 Gausian result. Thedifference between the two is taken and is converted to absolute value,which represents a sharpness image that can be used for the focus map.

In other words, a sharpness image is obtained for the macro-tile; and asynthetically created gray scale map that is the size of the macro-tileis made. The gray scale map has multiple areas that correspond to therespective tiles of the macro-tile. The incremental gray scale isdetermined according to the values in the sharpness image usinghistogram techniques.

If any of the bins in the histogram (which represent respective tiles inthe macro-tile) is above a threshold value, then the particular tile isnoted for high resolution examination and processing because thereappears to be adequate cellular material there. Furthermore, by taking afurther histogram of the absolute value of the difference between theoriginal transition characteristic of the macro-tile and the gray scaleGaussian filtered image, one can determine the particular maximum forthe tile; and such maximum may be used as a representation of theoptimum focus condition for the automated microscope for viewing theparticular tile during the high resolution pass. See block (250) in FIG.3.

The advantage of determining whether there is adequate sample on theslide for classification during the low resolution pass in themicroscope 12 is that such determination can be made relatively quicklycompared with the time needed to make the same determination using thehigh resolution objective 31. The advantage of determining, during thelow resolution pass, which tiles will require further examination in thehigh resolution pass, is to save the time needed unnecessarily toexamine tiles that do not have cellular material or adequate cellularmaterial there.

The scan pattern of the areas (tiles) of interest is made (255) andfocal information for each tile is correlated with position coordinateinformation for that tile from the position encoder 44 to provide afocus map (260).

All information necessary to perform a high resolution scan of the slideis now available. To commence the high resolution pass in the microscope12, the computer 20 operates the motor 53 to move the carriage 45 toplace the high resolution objective 31 into the microscope's opticalpath 29 (265). The lighting is automatically adjusted by the autoiris 52and motor 60 for the high resolution objective, and the stage 24 ismoved to bring the first tile or segment of the slide S intended to beexamined into the viewing field of the objective 31. The computer 20 acommands the autofocus 46 to adjust focus for the area, segment or tile(270) of the specimen under examination by providing an appropriateoffset signal on line 79 to the differential amplifier 77.

The camera 14 obtains a color video picture of the focused image (275),and that image is digitized and is stored (sometimes referred to asframe grabbing), as is described further herein. The stage 24 then movesthe next tile or segment into view and appropriate focal adjustments aremade in accordance with the focus map (280). If the last segment (tile)of the slide has been reached (285), a flag is set and processing of theslide will be discontinued after screening of that segment (290). Thestage 24 may wait at a location until an image of a tile is obtained.Preferably, though, such waiting time should be minimized to minimizethe time needed to examine a slide.

Preferably image processing of one or more than one segment or tile canbe carried out in the image processor 20 b simultaneously while an imageof another segment is being obtained by the camera 14.

The color components of the video representation of a segment are summedto provide a monochrome image (295), and that image is passed to theimage processor and digitizer 20 b where the primary classification ofthe segment begins. Initially, the image processor and digitizer 20 bperforms an adaptive threshold operation on the video image to enhancethe image contrast and eliminate noise from the background (300). Thisthresholded image is then down sampled to a manageable digitalrepresentation (305). The image processor and digitizer 20 b can thenperform erosion and non-connecting dilation operations on the digitalimage to separate the objects in the segment (310, FIG. 3b). Themonochrome and resultant filtered images are transferred to RAM of thegeneral processor 20 a (315, 320).

The erosion and dilation techniques are conventional image processingtechniques. They eliminate the effect of overlapping cells in which darkareas may appear due to the increased density of the overlap rather thandue to enlarged or especially dark cell nuclei. It is usually thedarkened nuclei or large size nuclei that are detected during theintegrated optical density (IOD) evaluation made in the low levelclassification procedure described further below. The erosion anddilation technique also enhances the accurate examination of the cellsduring the low level classification.

An object count also may be performed at this time to find out how manyobjects have passed the erosion and dilation. That number of objects isapproximately representative of the number of objects in the specimen.It is desirable that at least a minimum number of objects be included inthe specimen, for if the sample size is too small, then the test may notproduce meaningfully accurate or reliable test results. The object countor other means to determine the “validity” of the sample may be taken atother times in the described process of the invention. As is describedherein, it is desirable according to the preferred embodiment that aperson be the one making the tertiary classification; that personordinarily is expected to be reviewing images of several cells which theprimary and secondary classifications had determined to have arelatively high probability of being malignant or pre-malignant. It isdesirable that the person know that if the number of cells beingreviewed manually is zero or is relatively low, that is due to the factthat the other cells are healthy and not due to the fact that there werenot enough cells to examine in the specimen.

The processor 20 a then performs further primary feature extractionclassification on the segment such as with an integrated optical density(IOD) algorithm (325). Other morphological algorithms may alternativelyor additionally be used to classify based on features, such as color orfeatures relating to DNA ploidy analysis, immunohistochemistry, DNAhybridization, etc. These feature extraction algorithms isolate certainobjects which possess features typically known to be present inpathologic cells, such as a dark cell nucleus which is abnormally largein relation to the rest of the cell. The centroids of objects identifiedby primary classification as being possibly pathological are cataloguedand stored in RAM of the processor 20 a (330). If no objects have beenidentified (335), classification begins on the next segment (295).

Identified objects, i.e., electronic image representations of thosewhich have not been eliminated by the low level classification, aretransferred individually to the neurocomputer 82 as digital areas aroundthe object centroid (340). The neurocomputer 82 will perform secondaryclassification on the objects in accordance with its previouslycompleted training which is described more fully below (345).Additionally, an object count may be made at this point (and/or possiblyelsewhere in the process, as is mentioned above) to determine if theneurocomputer 82 has received a sufficient number of cells from theprimary classifier to indicate a valid test. For objects identified bythe neurocomputer 82 as suspect (350), the color representation of asuitable area surrounding the centroid is retrieved from disk andtransferred to the high resolution display board of the high resolutionmonitor (355). Cells having a classification less than the thresholdleve are discarded and the next centroid is obtained from the generalprocessor (340) and classified (345). When it is determined that alllocations, tiles, on the display board are occupied (360), the totalimage is transferred to the general computer 20 a for temporary storage(365) until all cells on the slide have been screened, and the highresolution board is cleared (370). Once all centroids found in theprimary classifier for that segment have been classified by theneurocomputer (375), the image for another segment is grabbed (275), andclassification for that segment is performed (295-375).

When all segments on a slide have been screened (380), the highresolution color images for the slide are transferred to disk (385) forstorage until a convenient time for display on the high resolutionmonitor and tertiary classification by a cytotechnician or cytologist.All arrays of suspect cells may be tagged in memory with informationobtained from the barcode to identify the slide on which they werefound. The location of the suspect cells on the slide also may bephysically marked on the slide by the slide marker 18. A reportindicating the test results for that slide, which is correlated with thebarcode information obtained from that slide, may be printed on theprinter 158 now or later.

The slide is then returned to the cassette 23, and another slide isselected (200) to begin the classification process anew.

It should be recognized that while the image processor and digitizer 20b, the general processor 20 a, and the neurocomputer 82 are describedand shown in FIGS. 3a and 3 b operating in a serial manner, in actualpractice as many functions will be performed in parallel as is possible.Consequently, the components 20 b, 82, 20 a may process different slidesegments or different areas of a segment concurrently, greatly reducingthe time required to screen a slide.

Turning to a more in-depth review of the classification method and withreference to FIG. 4, a block representation of the classificationfunctions of the screening device 10 is illustrated. Primaryclassification, such as low level morphological feature extraction, isperformed as indicated above by both the image processor and digitizer20 b and general processor 20 a and is represented in FIG. 4conjunctively as block 400.

Initially, as described more fully above, the video camera 14 obtains animage of the cytological specimen which is digitized for classificationuse. The primary classifier 400 first performs an erosion of the image.This erosion operation successively peels off layers of pixels from eachobject in the image so that all of the objects which are smaller in sizethan the smallest known pathological cell nucleus are removed from theimage. The remaining objects are then dilated, i.e., regrown, bysuccessively adding back layers of pixels to these objects, but they arenot dilated to the point where they are touching each other. The basicoperations of erosion and dilation can be found in several sources inthe prior art (e.g., Serra, J., “Image Analysis and MathematicalMorphology”, Academic Press, 1982).

Bsed on experience with an engineering prototype, it has been found thatfor every 1,000 objects found in a typical benign Pap smear no more thanabout 15 objects will pass the erosion/dilation screen. These relativelyfew remaining objects are then subjected by the primary classifier 400to an integrated optical density (IOD) screen.

Integrated optical density is the sum of the pixel grey values for eachobject. Pre-malignant cells tend to possess large dark nuclei. The IODthreshold is, therefore, set to filter out any object which passes theerosion/dilation screen but which has an IOD which is above or below thethreshold displayed by a truly pre-malignant or malignant cell. For the15 objects which passed the erosion/dilation screen, experience showsthat no more than ten objects will pass the IOD filter. Thus, theaverage combined filtration of erosion/dilation and IOD reduces an inputimage of 1,000 objects to an output image of ten objects. These tenobjects may include not only pre-malignant and malignant cells but alsoother objects with a high integrated optical density such as cellclumps, debris, clumps of leucocytes and mucus. The task of thesecondary classifier 420 is to distinguish the pre-malignant andmalignant cells from these other objects.

For the engineering prototype, classifier 420 was a backpropagationneural network hosted on an Anza Plus neurocomputer coprocessor boardresident on an IBM PC. The backpropagation network was trained with atraining set of several hundred known benign and pre-malignant ormalignant cells to associate a benign image with a diagnosis of 0.1 anda non-benign image with a diagnosis of 0.9.

In actual operation, the secondary classifier 420 is presented withimages passed to it by primary classifier 400. These are images whichmay be similar but are not identical to those used to train classifier420. The task for secondary classifier 420 is thus to generalize fromits training process to successfully classify benign and non-benignimages to which it was not previously exposed.

One major advantage of the present invention over the prior art residesin the fact that each image presented to the secondary classifier 420 ispre-centered by the primary classified 400 on the centroid of thesuspect cell nucleus. This is accomplished because erosion/dilation andIOD based filtration automatically results in a centering of eachsuspect image around its dark centroid. In prior art attempts to utilizeneural networks and other high-level template matching patternclassifiers for image recognition, difficulty has been encountered inconsistently presenting the classifier with the centroid of the imagerequiring classification. To use an example from another applicationdomain, backpropagation networks are excellent at reading handwrittenzip code digits but have difficulty in finding where the zip code is onthe envelope. The present invention overcomes this difficulty in thedomain of cytology.

In an engineering prototype, a 128×128 pixel image was stored aroundeach centroid which passed the low level filters of classifier 400. A64×64 window, also centered around the same centroid, was thencompressed using pixel averaging to a 24×24 pixel image. Note that thisimage is still centered on the same large, dark, image which passed theerosion/dilation and IOD filters of classifier 400. A set of severalhundred of these pre-centered 24×24 pixel images of known benign andnon-benign cells was used to train classifier 420. During feed-forward,i.e., post-training operation, when classifier 420 is presented with newimages it did not encounter during its training it must generalize fromthe training set images to select the diagnostic category which mostclosely matches the new image. A fundamental advantage of the presentinvention over the prior art is that during this feed-forward,post-training phase of its operation, classifier 320 is presented withprecisely the same type of 24×24 pixel images on which it was trained.These images are also centered on the centroid of the suspect nucleus byclassifier 400 in a manner identical to that used to prepare thetraining set images. This makes the generalization task of classifier420 far easier and thus much more successful than anything found in theprior art.

As noted above, classifier 420 is trained to associate a known benignimage with an output of 0.1 and a known pathological image with anoutput of 0.9. Such outputs represent, for example, the degree ofcertainty that a cell is normal or abnormal, respectively. Whenclassifier 420 is then presented with new, unknown cells, it generalizesfrom its training and attaches an output to the image. The closer thatclassifier 420 is able to categorize the unknown image into the benigncategory, the closer is its feed-forward output equal to 0.1.Conversely, the more closely that an unknown image appears to classifier420 to resemble the non-benign images of its training set, the closer isits feed forward output for that image equal to 0.9.

During testing of an engineering prototype in which a backpropagationneural network was used for classifier 420, it was found that no trulypre-malignant or malignant cell was attached to an output of less than0.75. In order to provide a margin of safety, classifier 420 onlyscreens out images with an output of 0.65 or less. Any image which isattached to an output greater than 0.65 is assumed to be a suspectpre-malignant or malignant cell. For each of these suspect images, theassociated 128×128 pixel image centered around its centroid is retrievedfrom image memory and displayed as one of a field of 64 such images on ahigh resolution output monitor for final classification by acytotechnologist.

All images which are classified by classifier 420 to have an output of0.65 or less are assumed to be benign and are not displayed on theoutput monitor. During testing of the above-described engineeringprototype, it was found that classifier 420 consistently filtered outover 80% of the benign images sent to it by the output of classifier400. In other words, over 80% of the truly benign images which pass theerosion/dilation and IOD screens of classifier 400 are assigned anoutput of less than 0.65 by classifier 420, leaving 20% of these imagesrepresenting suspect benign or pre-malignant and malignant cells to befinally classified by the tertiary classifier 440, i.e., thecytotechnologist.

As mentioned above, in an alternate embodiment the classifier 420 may bea high level template matching or holographic imaging filter. It ispossible to use these filters in an efficient overall processing schemebecause the object of interest has already been identified by the lowlevel feature extraction filter, classifier 400.

The overall operation of the cell classification system can besummarized with reference to FIGS. 5 and 6. FIG. 5 shows the screeningof a typical Pap smear which contains approximately 100,000 benign cellsand other objects. Through erosion/dilation and IOD filters, the primaryclassifier 400 will filter out 99% of these objects, passingapproximately 1,000 objects to the secondary classifier 420. Classifier420, which in the tested and preferred embodiment employs a three-layerbackpropagation neural network, in turn filters out 80% of these 1,000objects, passing the images of approximately 200 residual objects deemedto be most suspect of pathology to the output monitor for tertiary humaninspection 440. These 200 objects are assembled as two to three fieldsof 64 objects each. Each object is presented as a 128×128 pixel imagetaken from the video input to classifier 400 and centered around thesuspect cell nucleus. The tertiary classifier 440, the cytotechnician orcytologist, will then further screen the 200 objects to zero, since allwere benign.

The screening of a Pap smear having 50 pathological cell plus theapproximately 100,000 cells and other objects (classified above) isshown in FIG. 6. The primary classifier 400 will screen the slide downto 1050 cells (50 pathological cells and 1000 possibly pathologicalcells). The secondary classifier 420 will further screen these cells to250 cells (50 pathologic plus the 200 most suspect benign cells). These250 cells will then be screened by the tertiary human classifier 440resulting in 50 cells being classified as pathological.

The overall result is that instead of examining 100,000 cells under themicroscope, the cytotechnologist examines 200 to 250 cells presented ona high resolution color video screen, each screen containing 64 images,with the attention of the cytotechnologist focused on inspection of thecenter of each of the 64 128×128 pixel images.

In the preferred embodiment the invention is used to display imagesrepresenting the first 64 cells in the examined specimen which are mostlikely to be malignant or premalignant, i.e., they have characteristics,features, etc., of know malignant or premalignant cells. The actualnumber of images of cells, or number of cells themselves, may be more orless than the preferred number of 64.

Moreover, according to the invention such images may be presented to thecytotechnician one at a time, in an array of four, sixteen, or more orless images, and those images may be presented at various levels ofmagnification, which further facilitates and enhances accuracy of thetertiary classification.

The imae or images being screened may also be presented to thecytotechnician adjacent exemplary images which have features distinctiveof the conditions for which the screening is being performed. Forexample, if a slide is being screened for pre-malignant or malignantcells, one or more stored images having features common to known typesof pathological cells, such as a large, dark nucleus, may be recalledfrom memory and displayed adjacent the suspect image. Consequently, asthe cytotechnician screens cell images which have been classified by theprimary and secondary classifiers as suspect, he or she will be able toperform a side-by-side comparison of suspect cell images with a knownpre-malignant and/or malignant cell image. This provides a convenientvisual reference from which the cytotechnician can base his or hertertiary screening criteria. However, it is equally useful to employ theside-by-side comparison feature when the human classifier performs theinitial or sole screening of the slide.

The false negative rate of cytological screening is known to be afunction of the ratio of non-pathological to pathological objects whichare visually inspected on a daily, continuous basis. The presentinvention drastically reduces the number of non-pathological objectswhich require inspection from 100,000 to 200-250. In addition, allsuspect cell nuclei are pointed out to the examiner by their position inthe center of a 128×128 pixel rectangle. The result is a very much lessfatigued cytotechnologist and a very significant reduction in the falsenegative rate. It is axiomatic of cytological screening that thedetection of only one premalignant or malignant cell in a specimen issufficient to warrant a physician's further attention to the patientfrom which the specimen was taken. The converse, however, is not alwaystrue. The presence of only benign cells in a specimen does notconclusively mean that a patient does not have or is not developing theprecursors of cervical cancer. For example, a pap smear may not havebeen performed properly, and as a result the specimen may containinsufficient cellular matter for a reliable test or may be void of thetype of cellular matter in which cervical cancer most often develops.Whether there is sufficient cellular matter to comprise a sample sizelarge enough to possibly constitute a reliable test specimen may bedetermined, as described above, by an object count. A further method ofdetermining the reliability of the test, which may or may not be used inconjunction with the object count, is by ascertaining the presence ofcertain cells or types of cells in the specimen.

The lining of the uterus contains columnar shape endometrial cells, andthe vagina is lined with flat sheet-like cells, known as squamous cells.The interconnecting organ, or cervix, includes both these cell types andhas a transitional region called the squamo-columnar junction wherethese two cell types meet. The actual location of the junction withinthe cervix may move as a woman ages, becomes pregnant, etc., and variesamong women. It is in this transitional zone between squamous andcolumnar cells that cervical cancer generally develops first. Since themalignant or premalignant cells developing within the transitional zonemay not extend down into the vagina until cervical cancer has reachedits later stages, it is critical that the pap smear swab contact notsimply the vagina and lower areas of the cervix, but that it contact thetransitional zone further within the cervix. The area of the cervixabove the critical transitional zone, that area most proximate theuterus, is lined with columnar cells, known in the cervix asendocervical cells. Consequently, the presence of endocervical cells inthe specimen indicates that the transitional zone, or squamo-columnarjunction was, in fact, sampled.

Since even benign endocervical cells have many features that apremalignant or malignant cell possesses, the secondary classifier willgive these cells a ranking relatively higher than most other benigncells. Accordingly, for a specimen having no malignant or premalignantcells, the endocervical cells will be among the few highest rankedcells, and most likely among the highest 64 ranked cells. For thisreason it is desirable that the classification device display the 64highest ranked cells (i.e., those cells determined by the invention tobe the 64 most likely to be malignant or premalignant) or other objectsfor inspection by a cytotechnician, even when those cells are all rankedbelow that of a true premalignant or malignant cell. A trainedcytotechnician can easily distinguish an endocervical cell from otherbenign or malignant cells, and through the detection of least oneendocervical cell among the displayed cells it will be readily apparentthat the pap smear swab most likely contacted the transitional zone ofthe cervix. Consequently, a second method determining the reliability ofthe test is established. Further, in the absence of any premalignantcells or malignant cells, since an endocervical cell will be displayedamong the first 64 objects displayed if it is present in the specimen,the cytotechnician can identify the test specimen as adequate and thetest as reliable in very little time.

Another advantage had by displaying the 64 highest ranked cells is thatoften infections may be ascertained. Since cells that are irritated bymany infections, such as venereal diseases, etc., also display many ofthe characteristics of a premalignant or malignant cell, the secondaryclassifier will rank them relatively higher than benign, non-irritatedcells, while lower than truly premalignant or malignant cells.Therefore, for a specimen having no premalignant or malignant cells, theinfection disturbed cells will generally be displayed among the firstscreen of highest ranked cells for review by a cytotechnician. A skilledcytotechnician can then easily detect these cells among the relativelyfew (64) displayed cells, analyze them closer, such as by magnifying theimage or comparing the infected cell to stored representations ofinfected cells, if desired, and note the specimen as containing evidenceof infection.

Consequently, it is apparent that even though the classification deviceof the present invention may rank the specimen cells such that itusually may be determined from the first few cells displayed whether thespecimen contains pathological cells, it may be advantageous that acytotechnician examine all of the cells of the first displayed screen ofhighest ranked cells to determine the reliability of the test andwhether there may be indications of certain infections in the specimen.As will be evident to those of ordinary skill in the art, the presentinvention overcomes all of the obstacles to practical automatedcytological analysis inherent in the prior art. This is achieved by thepresent invention's unique combination of feature extraction andtemplate matching techniques. This combination of techniques overcomesthe requirement for image segmentation found in the prior art. Inaddition, this combination of techniques overcomes the major obstacle ofneural networks and other computationally intensive template matchingtechniques for the analysis of complex images such as cervical smears.This is achieved by the fact that the primary classifiers of the presentinvention result in an automatic centering of the suspect cell nucleusin the input array of the neural network.

By simultaneously overcoming the requirement for image segmentation andalso enabling the first practical utilization of neural networks forcytology, the present invention has resulted in the first practicalautomated cytological screening system. The present invention has beendemonstrated by an engineering prototype to successfully analyzestandard Pap smears with overlapping and partially obscured cells. Ithas also been shown to perform this analysis within the time periodtypically consumed by completely manual examination. Thus, the uniquecombination of techniques embodied in the present invention has achievedthe goal of over twenty years of prior art attempts at automatedcytological classification.

However, the invention is not limited to use as a primary screeningdevice, but may also be used to rescreen cells which a cytotechnicianhas already inspected and classified as benign. As such, the deviceprovides an effective check to focus the cytotechnician's attention tocells which may have been overlooked during the initial screening.

The invention can be designed and trained to classify other objects notexpected to be found in a typical Pap smear. Such an object may berepresentative, for example, of a herpes virus.

The invention may also be used in classifying unexpected cells on aslide which has previously been screened by a cytologist orcytotechnician on another basis. One such instance is in theclassification of a Pap smear taken for a post-menopausal patient. Sucha smear should not contain endometrial cells. Other instances arescreening for organisms, etc. Consequently, the slide will be analyzedby a cytologist for the presence of endometrial cells or other organismsand then screened by the invention for other unexpected cells orviruses, such as malignant/pre-malignant cells or the herpes virus.

Another important feature of the present invention is the ability toprovide adaptive learning for the neural computer 82. Such adaptivetechnique enables the neural computer 82 to be retrained or to betrained with additional information representing additional or betterdefined cells of specified characteristics. Characteristics of suchcells may be those of malignant or pre-malignant cells; may be those ofbenign cells; may be those of other types of cells or characteristics ofother types of matter expected to be found in samples intended forexamination and classification. Further, such adaptive training can becarried out in a laboratory or research facility and the results of suchtraining can be delivered to apparatus employing the invention in thefield.

It will also be appreciated that while the invention is described withprimary reference to Pap smears, the invention is applicable to most anytype of cytological specimen such as those containing exfoliative oraspirated cells, for example.

What is claimed is:
 1. A semi-automated method of classifying aspecimen, comprising the steps of: a) obtaining the specimen; and b)classifying the specimen to determine the likelihood that individualobjects in the specimen have attributes justifying further evaluation,said classifying including i) ranking individual objects in the specimenin an order according to the likelihood that an object has attributesjustifying further evaluation, and ii) identifying location coordinatesof one or more of the objects for review or further classification by ahuman.
 2. The method of claim 1, wherein the step of identifyinglocation coordinates comprises selecting coordinates of one or moreobjects according to said order.
 3. The method of claim 2, wherein oneor more objects associated with the selected coordinates is less thanthe number of ranked objects.
 4. The method of claim 2, wherein the stepof selecting coordinates further comprises the step of presenting imagesof objects corresponding to the selected coordinates.
 5. The method ofclaim 1, wherein said ranking step is performed by a neurocomputer. 6.The method of claim 1, wherein the specimen in a cervical smear.
 7. Themethod of claim 1, wherein said ranking step comprises assigning a valueto individual objects in the specimen, said value being on a scalebetween a first output value associated with a first condition and asecond output value associated with a second condition.
 8. The methodaccording to claim 7, wherein the first condition is benign and thesecond condition is non-benign.
 9. The method according to claim 1,wherein said classifying step further comprises selecting individualobjects to be ranked by analyzing at least one morphological feature ofeach object.
 10. The method according to claim 9, wherein the analyzedmorphological feature is selected from the group consisting of cellnucleus size and cell nucleus optical density.
 11. A method forproviding interactive review of objects in a specimen indicative of thehighest likelihood of abnormality in the specimen, comprising the stepsof: a) obtaining the specimen; and b) classifying the specimen todetermine the likelihood that individual objects in the specimen haveattributes of cell abnormality justifying further evaluation, saidclassifying including i) assigning individual objects in the specimen avalue according to the likelihood that an object has attributes of cellabnormality, and ii) selecting location coordinates of one or more ofthe objects to provide for viewing and further classification by ahuman.
 12. The method of claim 11, wherein said assigning step isperformed by a neurocomputer.
 13. The method of claim 11, wherein thespecimen is a cervical smear.
 14. The method according to claim 11,wherein said assigning step comprises assigning a value on a scalebetween a first output value associated with a first condition and asecond output value associated with a second condition.
 15. The methodaccording to claim 14, wherein the first condition is benign and thesecond condition is non-benign.
 16. A method of providinglocation-guided screening of a specimen for objects in the specimenhaving a likelihood of cell abnormality, comprising the steps of: a)obtaining the specimen; and b) classifying the specimen to determine thelikelihood that individual objects in the specimen have attributes ofcell abnormality justifying further evaluation, said classifyingincluding i) ranking objects in the specimen in an order according tothe likelihood that an object has attributes of cell abnormality, andii) identifying locations of one or more of the objects to provideviewing and further classification by a human.
 17. The method of claim16, wherein the step of classifying further includes selecting one ormore of the identified locations for review or further classification bya human.
 18. The method of claim 16, further comprising the step ofpresenting images of objects to a human corresponding to one or more ofthe identified locations.
 19. The method of claim 16, further comprisingthe step of generating a specimen map corresponding to one or more ofthe identified locations.
 20. The method of claim 16, wherein rankingthe objects comprises assigning individual objects in the specimen avalue according to a likelihood that an object has attributes of cellabnormality.
 21. The method of claim 16, wherein said ranking step isperformed by a neurocomputer.
 22. The method of claim 16, wherein thespecimen is a cervical smear.
 23. The method of claim 16, wherein saidranking step comprises assigning a value to individual objects in thespecimen, said value being on a scale between a first output valueassociated with a first condition and a second output value associatedwith a second condition.
 24. The method according to claim 23, whereinthe first condition is benign and the second condition is non-benign.25. The method according to claim 16, wherein said classifying stepfurther comprises selecting individual objects to be ranked by analyzingat least one morphological feature of each object.
 26. The methodaccording to claim 25, wherein the analyzed morphological feature isselected from the group consisting of cell nucleus size and cell nucleusoptical density.